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r ' ^ -1- DUPLfCATE 

Image Stabilisation System and Method 

This invention relates to a system and method Ibr the stabilisation of video 
Images produced by a camera. In particular, it relatea to means for reducing 
or eliminating the offset of successive Images recwded by the camera caused 
by undeslred camera motion such as wobble. 
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undeslred camera motion can distort the received Image, or make visual 
analysis of the Image hard, due 1o the perceived Jumplnesa on the 
reproduction apparatus. Such camera motion can be caused, for example, by 
wind blowing on a camera support pole, or unstable support due to the 
camera being hand held, or mounted on a moving vehicle or boat efts. 

One method used to stabilise the video Image from a camera suffering 
16 unwanted movement is to use a motorlsed camera mount that measures 
camera movement caused by such things as wind buffet, and physically 
maves the camera in response. Gyroscopes may be used to detect the 
mayement of the camera, and electric motors used to con^ctfbr It This 
provides, in effect, an electromechanical negative feedback loop that 
20 attempts to keep the camera In a f«ed position. This solution can be very 
effective where the camera is used in surveillance applications where the 
camera Is mounted on a permanent or semi-permanent mount The mettwd 
is able to reduce bluning of the video image even when very skjw camera 
shutter speeds are used as. in a correctly stabilised system, the camera itself 
26 is not moving. It is also unaffected by the lighting conditions of the scene, as 
no reference is made to the recorded video signal, but the technique can be 
costly, cumbersome and may require significant electrical resources, 
especially if ttie camera system is laige. 

30 A developmerrt of tills is to use sensors to detect the camera motion as 
before, but use the signals f^om m sensors to process the signals produced 
by the camera. The processing may Involve electronically shifting the image 
with the Intention of bringing ft Into alignment with preArfous Images recorded 




by the camera This approach eliminates the requirement for a motorised 
camera mount but the movement sensors are stiil required. 

image stabilisation can be done purely by eiecfanonic processing of the image 
5 signei. These methods consist of comparing a current image produced by the 
Camera with a reference image, and spatlaliy moving the current image so as 
to bring K into line with the reference image. Different techniques are 
employed to do this. 

10 One such techniquet GB001 2349,7 describes a method for stabilising the 
video images of a camera. This approach uses a g]ofc)al motion estimation in 
which explicit horizontal and vertical components are used as a measure of 
how much a current image needs to be shifted to provide a best match 
against a reference image. These components are high-pass filtered before 

1 5 being Input to the processing, so allowing slow camera movements to go 
. through to the output video signal, whereas sudden movements are Input to 
the stabilisation processing. This system Is susceptible to errors if something 
in the scene at which the camera is pointing is mo^^ng* and It has no facility 
for coping with deiibeiate panning or zooming of the camera. 

20 

According to the present invention there 1^ provided a video image 
stabilisation system that is arranged to receive one or more signals 
representative of a plurality of Images wherein^ for an image n following at 
least an image {n -1) and an image (72-2) the system is arranged to estimate a 

26 Global Motion Ofi^et (GMO) value between image n and a previous image 
representative of the spatial eeparation between the image n and the previous 
image, and apf^y a corrective nnavement to the image n based upari this 
GMO, characterised in that 

the system is arranged to estimate the GIVIO for the Image » with 

30 relerence to a masic that represents a r^ion or regions of fiie image n that are 
not to be oonsfdered in the GMO estimatioiii, the rdgion(6} being region(s) 
eetbmated ae iilcely to mieiead the calculation of the GMO. 
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The present Invention Is able to reduce the effect of unwanted camera 
movements under some circumstances. It provides a system for the reduction 
of the effects of camera movement on the resultant video Image. The system 
uses previous Images (n the video sequence as a reference when calculating 
5 the movement of the cument Image. 

The GMO is a measure or estlmalion of the distance of the current Image » 
fiom the reference Image r. This distance is ^ically a vector oomprisina the 
number of pixels that » is fttjm r. In both the horizonial and vertical planes; l.e. 

10 It indicates the shift that should be applied to » so as to get a best match 

between n and r. and Is calculated by any suitable means. Note that different 
means for calculating this beat match may produce different results, not aU of 
them being ttie optimum value. Calculating the optimum value may be too 
time consuming, or require too much computer resource, and so other 

15 techniques may be employed that gh/s an approximate value for the offset. 
The reference image r Is preferably the previous Image in the video sequence. 
I.e. image in -1). although other Images may be used. 

Further infonnatfon Is used in estimating the GiWIO to get Improved 
20 perfbnnance. A mask is one such piece of infonnation. This Is a binary 
image that Is used to exclude pixels In the image n that may adversely affect 
the GMO calculatbn. The mask Is preferably generated by examining the 
Image n for objects movbig within sub-regions of the image, and then setUng 
the corresponding bits of the mask to exclude these areas. This is preferably 
25 done by dividing the image « into a plurality of sub-images n « and calculating 
a Local Mofion Offset (LMO) for each sub-image n g. The LMO may be 
. calculated in the same general manner as the GMO, but different techniques 
may be more suitable due to the fact that each sub-image « s is smaller than 
the image «. The LMO may be calculatad using a conespondlng sub image 
30 taken from the same reference image r as used with the GMO. but preferably 
the corresponding sub-Image taken ftom Image (n -2) is used. Again, other 
rafsrence Images may be used. 



r 




The mask Is preferably augmented to correspond to area$ of the image n 
which are represented by pixels that do not behave with a desired 
characteristic. These pixels may be ''dead" pixels, which appear to be always 
on or always off, or could be pbeels that behave in an Irregular manner to the 
5 Inoomtng light They may also be pixels that are deemed to be corrupted by 
noise above some threshold. The detection of noise may be done In any 
suitable manner. Such areas are known herein as anomalous areas, and the 
pixels making up the areas as anomalous pbcels. 

10 The images used to calculate the GMOs and LMOs are preferably sub- 
sampled before the oaiculation takes place. This has benefits In that the 
calculation effort is reduced, and the low-pass filtering Inherent Fn the sub- 
sampling process makes the stabilisation system more rssillent to image 
noise. Local minima in the calculations are also less likely to be a problem. 

15 For Improved accuracy, the GMO or LMO calculations may be iterated at 
multiple resolutions, starting at a lower resolution, generating a GMO/LMO 
from this, and then moving to a higher resolution taking account of the 
GMO/LMO calculated in the previous iteration. Multiple iterations at a given 
resolution may also be done before moving to a higher resolution. 

20 

The GMOs and LMOs calculated for the input images may advantageously be 
used to estimate whether a pan or zoom operation has been applied to the 
camera. Herep pan is taken to mean movement of the oamera such that it 
points In a different direction, either horizontally or verUt^IIy, or both^ and 
25 zoom is taken to mean that the fbcal length of the camera lens is changed 
such that a different field of view is seen by the camera. 

Usafully, a prediction of motion ofket emirs can be made using thd U\40s and 
GMOs. One Such error Is lock-on". This may occur when, for example* the 
30 substantially the whole seme visible by a camera is taken up with a moving 
object For example, if the camera were pointing at a road scene, and a very 
large lorry were to pass ciose to the camera, then much of the innage recorded 
by the camera may t>e taken up by the lorry. Without any emor correction, this 



would give the Impression of a sudden pan taking place, which would cause 
the stabilisation routfrie to emoneously try to track the movement 

Another such error relates to "static camera detedion". Analysis of the GMO 
history Is used to predict when the camera is not moving. If this state Is 
detected, then the GMO vector is set to zero. Without this, the accumulation 
of errors in small GMOs can lead to enoneous stablllsatton to occur. 

The GMO vector Is preferal»ly translated Into a final stabilisation offiset (SO), 
whteh represents the vector to be appfled to the cunent image after 
calculation and processing of the GMO vector has been carried out. This 
translation preferably takes Into account the motion offset emirs and 
estimates of pan and zoom operations. The translation preferably Involves a 
decay factor that tends to reduce the influence of SOs applied to previous 
Images, This Is useful, as respective SO vecfors tend to accumulate, such 
that an offset applied to an Image may othenvlse remain even if the 
requnem^t for such an offeet has ended. 

A video signal that has been processed according to the current invention 
may result in the edges of the video image not being aligned with the edge of 
the display area of a display device such as a television screen. This is as a 
result of the image being shifted relative to the display device according to any 
detected motion offeets. Preferably, such edges, which may be rapidly 
changing in position, are kept hidden from view by means of the addition of a 
border area bebween the Image edge and the display area of a display device. 
More preferably, the bonJer is adapted to change size according to the 
displacement of the images relative to the display area. This may take into 
account the displacement of prior images as well as the one cuirently being 
displayed. 

Alternatively, any blank areas around the Image to be displayed caused by 
shifting of the Image may be augmented with image Information from prior 




images. In this way, a full Image having no artificial bonders can be presented 
to the display device. 

As a further alternative, the image to be displayed may be expanded in size 
5 such that any blank areas are flOed. This may be done by scaling the Ifnage 
using imown algorithms. 



1 0 Note that the video signal or signals fripul to the system may come directly 
from a camera, or they may tsoma from some other means such as a video 
recorder or digital image storage on a computer system, or a mixture of such 
sources. 

15 The present inventton may be implemented on a computer system. Including 
those incorporating a general purpose microprocessor, and those 
incorporating a Digital Signal Processor device. A computer can be 
pnogrammed to so as to implement an image stabilisation system according to 
the current invention. 

20 

The invention vwll now be described In more detail, by way of example only, 
with rel^rence to the following Figures, of which: 

25 Figure 1 diagrammaticaily iiiustrates the hardware upon which the ounent 
invention may be implemented; 

Figure 2 is a data-flow diagram that shows the top level operation of an 
embodiment of the current invenfion: 

30 

Figure 3 shows In mora deteiB the step of estimating the image motion offsets 
and Local i\4otiQn Masks; 
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Figure 4 shows in more detail the calculation of the GMO for each image 

Figure 5 shows In moi« detail the calculation of the l^Os fbr each Image m 
5 Figure 6 shows In more del^l the process <rf generaUng the maek image; 

F^ure 7 shows In more detail the steps Involved In correcBng Itor motion offeet 
errors in the GMO; 

10 Rgure 8 shows in more detail the operation of camera pan and zoom 
detection; 

Figure 9 diagrammatlcally illustratss the use of the outer set of LMOs for an 
image n in the detection of a zoom operation; 

15 

Rgure 10 shows in more details the steps Involved in generating a final 
stabilised Image given the previously calculated information: and 

Figure 1 1 shows the effect of the dynamic border generation on the slabHised 
20 image. 



Figure 1 1llustrates a typical hardware arrangement that can be used to 
impiemant the current invention. In this example the video signal is generated 

25 by a video camera 1 mounted upon a mount 2. The camera is subject to 
buffeting by the v»rind. which. If sufficiently strong, will cause the camera to 
wobble on the mount, as niustrated by the airows 3. The camera supplies a 
video signal to the stabnisation system 4, the output of which is a video signal 
that has been processed as described herein, which may then be displayed 

30 on a suitable display 5 or recoided on some suitable medium. The invention 
may be applied to many different video signal formats, both digitel and. with 
suitable digitisation, analogue; the cument embodiment is set up for 
processing PAL and NTSC signals. 
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The stabilisation system 4 carries out tti© processing of the signal to attempt 
to reduce any instability of the camera image. The system 4 corvtalns an 
analogue to digital converter (ADC) 6 that digitises the incoming analogue 

S video signal. The digital signal is then fed to a signal processor 7. This 
processor 7 Is able to perform complex calculations and manipulatione upon 
the incoming data stream and provide an output signal that may be converted 
to an analogue signal by the digital to anatogue converter 8 ready for replay 
on the display unit 5. The processor 7 is connected to a digital framestore 1 0 

1 0 that is able to store the current image n ftom the camera, as well as the 
previous two images (m-2). These are used in the processing. The 
processor 7 Is also connected to a general digital memory 9. This memory 9 
holds the program that Implements the current invention, as well as t>elng 
used as a general storage area lor data generated in connection with the 

1 5 operation of the invention. 

Tiie ADC 6 digitises the incoming signal at a resolution of 720x288 (for PAL) 
or 720x240 {for NTSC), although only the central portion of this is used as an 
input to the processing algorithm, as the outer parts of the image may 
represent parts of the scene that are not present in tMo successive images 
due to the camera movement Also, certain camera types contain inactive 
pixels at the borders of the Image area. The active area used in the cunreni 
embodiment has a resolution of 576x256 (for PAL) or 676x224 (for NTSC). Of 
course, the results of the processing are applied to the whole of the digitised 
Image. 

Figure 2 shows a top level data-flow diagram of the operation of one 
embodiment of the current invention. This assumes that at least two images 
have been pre\^ou6ly captured, and that the cun^ent image n has been piao^ 
30 in the framestore 10. This is going to be the case tor aD occasions apart from 
the time when the system is first svt^tehed on. 
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The processing relating to image n is as follows. Images n 1 00, (r-1 ) 101 and 
{b-2) 102 ar© presented to module 1 04 which calculatee the GMO and LMOs 
for Image n, as well as a mask image . Details of how these offeets and mask 
are calculated are provided later. The Image is divided up Into 6x8 (tbr PAL 

5 inputs) or 6x7 (for NTSC) fegions for cateulatlon of the LMOs, although of 
course a different number of regions may be used. Note that the mask may 
be referred to henain as a Local Motion Mask (LMM). as its primary task Is to 
mask out areas of the Image where local movement has been detected. The 
mask may, however, be set to Include anomalous areas or pixels. The mask 

10 may also be a composfte mask, derived from any local motion detected and 
anomaloua areas, although it may still be referred to ^ an LMM. 

Following calculatbn of the GMO, LMOs and the mask, this embodiment of 
the inverrtton analyses these values and previous such values to check for 
1 5 motion oRiE^t em^rs. This is done in module 105. and produces "error 
correcfied'' offsets GMO^(if) and LMOsecCr)' The data is used to defect far 
lock-on enotB and static camera wravs. The details of how this is done is 
described In relaOon to Figure 7 below. 

20 The cunent embodiment then examines GMOb:(r) and LMOsEc(»)a8 modified 
in module 105 to detecl^ and compensate for. desirable camera movements, 
namely pan and zoom. This detec^on produces a 'camera state" value, which 
is then used to adjust how the calculated oiiiBet is applied to the image n. The 
detectioft is done in a camera model module 106. Details of how the pan and 

25 zoom states are detected are provided below, with referenoe to Figure 8. 

The state outputs of module 106 and the GMOedin) value calculated in 
module 105 are now used to calculate the final stabiiisation offset to be 
applied to Vne image it. This is done In module 107, which produces an output 
30 offset SO(n). This Offset is passed to the dbplay routine module 1 08, whteh 
shifts image n according to ttie value of SO(n), and sends the signal 
representative of this image to a display device or recording mediumt after . 
first applying any bonders as desorlt>ed below. 




Figure 3 shows In more detail the process of generating the GMO, LMOs and 
LMM Ibi* each image n. as used in the current embodiment Images ». 
and (ii-2) 100. 101, 102 are supplied to module 104. Images n and {n-l), and 
5 the LMM calculated using lmage(R -1 ) in the previous Iteraifon are used to 
estimate a GMO In step 109. based upon a known algorithm described later in 
this spedficatlon. with reference to Figure 4. Images («) and (n -2) are also 
used to estimate the LMOs for image n, in step 110. For this, the image n fs 
divided up into an anay of 6x8 (for PAL) sub-Images, or local Images, and an 

10 LMO estimated for each one. The last-but-one Image (« -2) i$ used in the 
comparison as the difference between this and the current image is lil<ely to 
be greater than if the last image is used, leading to better detection of 
movement within the local sub-image - it has been found that local mottan 
tends to be smaller than global motion. The algorithm used to calculate each 

15 LMO fs quite similar to that used to calculate the GIMO, and is described in 
more detail later, with reference to Figure 5. 

The LMOs generated for image /i (represented as LMO(K))are used, along 
with the <3M0 for Image n (GMO(b)). to generate a mask. The mask Is 

20 generated in module 111. This mask is used bi the estimation of OMO(fi-i- 1 ) 
and has 1 bit of Infbnmatlon for each of the pixels in image «. If this bit is a 0. 
then the comsspondlng pbcei in the Image (n +1) b not used in the calculation 
of QMO(n +1), If the bit is a 1 then the comesponding pboel In the image {« +1) 
is used In the calculation of GM0(n+1). The mask Is stoi^ In memory 1 12 

25 untfl it Is to be used. The mask Is used to mask out those areas of the image 
where local motion — which would otheiwise distort the calculated GMO value 
— has t>een detected, and is also used to n^sk out anomalous areas. An 
^fimate 1 14 of tt» level of noise in the inrage is carried out, and stored 112. 
1t>r later use More detail cf the mask generation and noise e^mat'on is 

30 provkf^ later, with regand to Figure 6. 

Rgure 4 shows In more detail the steps taken In module 1 09 in tfie generatioi 
of the GMOs. The Inputs to this module dre the cunent image n 100 and 
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previous image (yi-l) 101. the previously calculated LMM 113 and the noise 
level estimate 114. There are three main steps to the procedure: 

In the first, the input images and the LMM are sub-sampled in modules 
115 to reduce their size, and hence inoease the speed and noise resilience of 
5 the operation; 

The second does the main calculation of the GMO, in module 116, 
based on the sub-sampled image; 

The third conBcts the calculated GMOt in module 1 17, to take account 
of the sub-sampting. 

10 

The sub-sampling of the Image Is done udng a tNlfnear interpolation process, 
which w9l be familiar to a person skilled in the relevant art, and will not be 
described further herein. Mare details of this can be found in iSonka, HIevac 
and Boyle. Image Proce^ng and Machine Vlston\ 2"** edition, 1998 (PWS) 

1 5 Brooks/Ool© Publishing. The current embodiment sub-samples to reduce the 
resolution by a fectDr4 in the vertical dimension and by a factor 8 In the 
horizontal dimension. This has been found to produce adequate results whilst 
giving a useful reduction In computation time. Other fc>ene1rts of eub-sampiing 
include a lower susceptibilit/ to noise in the frnage, along with a reduced 

20 probability of the GIVIO calculation being confused by local minima, as the 
sutKsampIIng eflectively low-pass filters the image. 

Ths calculation of the GMO value involves calculating the translatton Vhst 
needs to be applied to image » so as to minimise tiie misnsgtetratton of the 
25 image n with image {n -1 ), as sub-sampled. A measure of this misregisinatton 
is found by summing the intensity dflfPerences pbcel-by-pixeJ between the 
Images n and (n -1 ) (here represented as I„ and for clarity), to cfeate an 
error value E, 

30 The procedure attempts to minimise the square of this error value. Thus, 
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where lj,(x^) represents a point ^.^wlUitn Image n. and endgCx^y) are 
transfonnations of fhe cx>-ordinates x and y (Le. each pixel location) 
respectively that transform the Imsge co-ordinates of Innage n Into those of 
image {n -1 }. A Taylor expansion of Eqn 1 yields an equation that te 
5 conveniently analysed, and by means of dHferentiating this with respect to the 
parametens of the transformation functions /and g and setting these to zero, 
the resulting equation may be solved to reveal the latest update to the 
transformation, or GtAQ. Note that the Invention pre-processes the images 
before doing the GMO calcuiation to provide a more accurate result. This pre* 
10 processing involves multiplying on a pixel-by-pixel basis the image (as sub- 
sampled) with the mask (as similarly sub-sampled), effectively reducing the 
active area of the image n which has the effect of improving the accuracy and 
simplifying the calculations. 

15 The use of a Taylor series approximation of the error equation, Eqn 1 

introduces appn^ximafion en-ors, as the current embodiment only uses the first 
order tenn. This can result in the process finding local minima of the emor 
function, and hence providing an incorrect GMO. To reduce the effects of the 
approximation, the process of calculating the GMO described above is 

20 iterated. An initial motion estimate 200 of 0.0 is used, which is updated at 
each iteration. Each iteration uses an updated version of the cunrent image n, 
warped, or shifted, by the latest estimate 201 of the transfonDatlon 
parameters (i.e the latest value of the cun^nt GMO being calculated) 

2S in calculating the GMO tiie current embodiment also uses an annealing 
process to improve accuracy and help mitigate against enrors caused by 
objects moving through the scene. The annealing process decides whether 
any given pixel Is to be used in the current GMO calculation iteration, it does 
this by looking at the absolute difference between each pbcei In the Image (n- 

30 1) and the con^espondln^ pixel In the Image n that has been warped or shffled 
by the current value of the GMO being calculated. If this difference is greater 
than a threshold value then ft Is not used In the next iteration. The noise level 
estimate 1 14 is used In calculating the threshold value. This process 




DEC-2002 15! 50 hKUH ir hmlvctu^ 



13 



excludes pixels that do not line up despite having been warped - the cause of 
this beins mostlikely dueio imaoe anomalies or movement of objects through 
the scene. As the Images become mons aligned due to the iterations of the 
process, the threshold chosen Is decreased, as more of the pixels in the 
5 images should line up resulting In a reduced absolute difference value. This 
processimplementsafomiofrobuslstatlsticalestimation. Other such robust 
statistical estimation methods are known and are applicable to the current 
invention. 

10 The current embodiment calculates only the translation of the Image « that 

provides an Improved registration. Hence, for this case. 
. j^^y^.^^^^ and Rotational and scaling enors are not 

currently considered but the Invention may equally be applied, with suHaWe 
adaptation, to stabilise a video signal derived from a source that may be 
15 susceptible to rotational or scaling instabilities. This adaptation Involves 
representing the funcHons/^^cy; and g(x,y) in terms of translailonal. rotational 
. and scale parEmmeters thus: 
f(x,y)=^+bx-cy and g(x,y)=Ay^cx+by, 

where the scaling factor = (b'Wf and the degree of rotation = tctn'(cA,). 
20 These equations are then solved In a similar fashion to that described above. 
More Infomiatlon on this and other aspects of the calculation of the GiVlO may 
.. be found m Kent P. TWiiiHfsso/uffon Image Registration'' /EE Colloquium on 
, Muttiresomtkin Mo^efiing and An&iysls in image Processing and ComputBr 

VTsfOn. 1995. and In Kent. P, "MulSmsoluHon Image Registmtion and 
25 htosJdn^ Journal of Defence Sc/ence. Voi. 1 No. 2, the contents of both of 
which are hereby Included by iBference. Note that these references detail a 
. multiple tBSOluilon technique, whereby the GMO value calculated at a lower 
lesolutlon is then appfied to a subsequent GMO calculation perfomned upon 
an Increased resolutton version of the Image. This can be repeated as 
30 necessary to get the required accuracy. The cunent embodiment has been 
found to give satisfectoiy results with a calculation peifomaed at a single 
resolution, but may he adapted to use multiple nasolutlons if required. Other 
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methods of calculating the GMO and LMOs exist and are applicable to the 
current invention. 

Foilovring the estimation of the GMO as calculaled above, the value obtained 
5 Is multiplied 11 7 by the same factors used In the sub-sampling to take account 
of the change In nssolution of the bnage. Note that the current embodiment 
. uses sub-sampled versions of the image n only fbrthe calculation of the GMO 
and LMOs. All other operations work on tfte Ima^ In Its original resolution. 

10 Figure 5 shows In more detail the process used to calculate the set of LMOs 
for each image n. The process is based on a simplified version of that used to 
calculate the GMO as described above. The image n is again sub^ampled as 
before to reduce the woridoad and image noise. After this, the sub-sampled 
Image Is divided up, in modules 120. into 6x8 sub-image blocks (for PAL). 

15 each of size 12x8 pixels. Each sub-image block Is passed to a simplified 

version 121 of the GMO estimation routine, which lacks both the masking and 
annealing functions. The calculation Is done iteretively as beford^ using an 
initial motion estimate 202. which is updated at each pass, as indicated by 
numeral 203. The masking and annealing ftinctions are not needed due to the 

20 small size of each sub-fmage being processed. The vectors produced by 
module 121 are then multiplied up in module 122 by the same factor used in 
the sub-sampling to account for the reduced resolution of the images used in 
the processing. The current value of the GMO for Image and for Image {n — 
1 ) are then subtracted from the calculated UVIOs values. This ensures that 

25 the LMO values are not con-upted by camera movements. The resulting 

LMOs 123 are vectors that hold the horizontal and vertical shift required to get 
a best mateh between the each sub-image of n and the conresponding sub- 
image of (ff-2). 

30 F^ure 8 shows in mors detail the process of generating the masic This mask 
is used as described above to remove frmi the relevant calculations pixels 
connected with both focal motion effects and pixels behaving anomalously. If 
a sub-Image has an LMO greater than a given threshold then the regton 
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correspondlng to the whole sub-Image Is masked off by setting the 
appropriate mask bfts to 0. This mask is the LMM, and is calculated in 
module 125. The LMM, In this embodiment. Includes masking that 
corresponds to i^ions of the Image a In which anomalous pixel behaviour 
has been detected. 



The anomalous pixels are found in the following way. An Image representing 
the absolute difference between image it as shifted by the current QMO. and 
10 Image {nA) is produced 124, I.e. \M = !(«)- IM)- "^^e parts of the um 
mask due just to motion effects as derived above is also used, such that Mn) 
only comprises those sub-Images where signlflcanl tocal motion has not been 
detected. 

1 5 The intensity levels of the resulting difference Image are then examined In 
module 126. This is done by first generating a distribution of the pixel 
Intensities of [M- A threshold is then set, as the lowest decile of this range, 
multiplied by 5 - this factor having been chosen empirically to give a 
reasonable degree of robustness to motion anomalies. All pixels above this 

20 Intensity are then regarded as anomalous, and so the con^espondlng bits Iri 
the LMM are set to a zero, in module 127. to exclude them from the relevant 
operations, as shown in Figure 3. This anomalous pixel threshold Is used as 
the noise level estimate 1 14 used tn the calculation of the GMO. 

25 Figure 7 shows a data-flow diagram of the motion offeet error correction of 
module 105. For this purpose, module 105 has access to the GMO vectors 
from the cuiwnt 1 18 and previous images 128 and the set of LMO vectors 
123 from the current Image that are stored in the system memory. Module 
105 first does a tock-on estimation 129 to check for distortion of the GMO 

30 caused by very large moving objects. It does this using GMO(k), LM0$(»), 
and GMO(»-1)to GMO(»-25) (PAL) . orto GI4O(ii-30) (I^SC). which have 
been stored In memory previously. 




A lock-oh IS deemed to have ocoirred when: 

i- GMO(a-1 ) to GMO(w-25) (or GMO(b-30) as appropriate) are all less 
than a given threshold value (5.0 pixels in distance Is cun-ently used); 
ii. GMO(ff) is greater than the given threshold; and 
5 III. IMore than 50% of the lM06(n) are greater than a given threshold (4.0 
pixels in distance is currently used). 

The current embodiment only looks at the horizontal component of the GMO 
and LMO vectors, although it is of course possible to use either or both 
1 0 components. When a bck-on is deemed to have occurred, the horisxintal 
components of the 6l\40(/s) and LIUIOs(k) are set to »n3. Thb has the ^ect 
of stopping ail stabilisation effects for this axis, for this pari:icular image in the 
sequence. Stabilisation in the vertical axis can stiii occur however 

1 5 Module 105 also analyses the current GMO and the GMO recent history to 
estimate 1 30 whether the camera is actually stationary, it does this by looking 
at approximately one second's worth of the GMOs, equating to GMO(«) to 
GMO(re -49) (for PAL signals). If all examined GMOs are less than a threshold 
- usually set to 1 pixel for both the horizontal and vertical axes - then the 

20 camera is deemed to be static. When this Is detected, the current horizontal 
and vertical component of GMO(n) is set to zero. effecUvely disabling all 
stabffisation for that image. 

The error corrected offiBets from module 105 are deemed GMO^^ 130 and 
25 LMOSec131. 

Figure 8 shows in more detail the operation of detecQng the camera state 138 
» i.e. wh^er a pan or a zoom operation has fal<en place. In detecting a pan, 
the difference between two GMOec sums fs calculated 133. One le the sum 
30 1 34 of the GIM06ec(») from GMOec(1 ) to GIMOec(r) 0^^- the accumulation of 
GMOec values since the system was switched on). The other 135 uses the 
same offsets passed through a h^lvpass filter before summation. The filter 
used is a second-order Bessel filter v\rfth a cut-off frequency of Q.SHz, although 

i '. ' " • ■ • . . ' ' . : 
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a person skffled In the Wievant arts wfH realise that there are many filter 
characteristics that will be suitable. More details of the calculation of this filter 
charactaristte can be found In Rabinsn LR., and B. GoW, Theory and 
Application of D^Mf Signal Processing'', Prentice Has. 1975. pp22S-23Q, A 
5 large enough difference between these sums indicates the presence of 1^^^ 
frequency global motion, typical of a pan. Note that the pan defcecUon 
described above Is similar to that which would be achieved by low-pass 
filtering the GMOk sum; however, the above method is used as the high- 
pass filtered values are used In other processing, and so are already present 
10 in the system memory. The effort of calculating the low-pass filtered values is 
thus saved. 

If the dlfierence between the sums exceeds a threshold (set Id 50.0 pixels In 
this embodiment), then a pan is deemed to have occurred. Until this threshold 
15 Is exceeded, the pan detecttan shows a No Pan State. The first time the 
threshold Is exceeded, the pan detection shows a Possible Pan State. If this 
happens for several consecutive images (set to 30 images In this 
embodiment) then there is enough evidence of a pan and the pan detection 
shows a Pan State. 

20 

once the pan detection shows a Pan Slate. It will continue to show a Pan 
State until the difference between the sums does not exceed the threshold. To 
smooth the transition, the pan detection will show the Possible Pan State for a 
few images (set to 30 images In this embodiment) before returning to the No 
25 Pan State. 

In detecting a zoom operation 136. the LMOsec(«) 132 from a group of local 
motion blocks centred around the centre of the Image are examined. Typically 
this is a rectangular border, one blodcdeep around the edge of the grid of 
30 local motion bloclts. This is iilusiraled In Figure 9. Here, an image «, is shown, 
represented by the la,^ rectangle 11. An InnerporBon 12, Is shown divided 
up into a set of 6x8 local motion blocks, eg 13. Azoom-in isdetectedlf 
LIVIOSec ftom selected blocks appearto show movement generally towards 
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the centre of the Image. Likewise, a zoom-out is detected If the movement is 
generally away fipm the centre of the image. The selected blocks are those 
on the border (e.g. 14) of the Inner portion 12 image n. The blocks on the left 
and right side of the recfengle are e}^mined to See if they show motion 
5 greater than some threshcdd in the horizontal axis. Similariy, the bioci<;s on the 
top and bottom of the rectangle are examined to see if they show motion 
greater than some threshold In the vertical axis, it will be seen therefore that 
comer blocks contribute to both the hori2X>ntal and vertical analysis. 

1 0 For each block in the gix>up, the magnitude of the motion offset is compared 
against a threshold value. If a block has a motion offiset component magnitude 
greater than a given threshold then that block is cx^nsldered to have 
significant motion. 

1 5 For each block that has significant motion, the direction of the motion relative 
to the centre of the image Is used to Judge whether that motion shows a 
zoom-tn orzoorrhout. All blocks within the group are then examined to decide 
if a zoom operation is in progress* In the current embodiment, a zoom is 
deemed to have occurred if the following inequality is saQsfied: 

20 M^z(^y -^zi^^hjN, {Eqn 2) 

where Nza^i) is the number of blocks in the group indicating a zoom in; and 
Nzrouo is the number of blocks in the group indicating a zoom out, 
and iVia is the tota! number of blocks in the group. Of course, the direction of 
. the zoom can be found by comparison of N'zfoo and Nzfouu, 

25 

The inequality (Eqn 2) may in fact calculated bvice. It is first calculated with 
values for iVieyai) arid Nz(t^Q which include only those blocks where a motion 
offset component greater than 5 pixels occurs. If the inequality Is then 
satisfied, then a 'Tast zoom" is deemed to be occurring. If the Inequality is not 
30 satisiied then tie calculation is repeated, this time Including in Nzcia) and Nzc^mo 
those blocks where a motion ofliset component of 1. or more pbcels occurs. If 
the inequality is now satfsfled then a "slow zoom" is deemed to be occunlng. 
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The reason for classifying a zoom as either a fast or slow zoom is because it 
has been found that bettor stabilisation is achieved by handling them 
differently. The difference in handling for the two stateis is given below. 

5 if a zoom state Is detected for more than a number of consecutive images (2 
for a fast zoom and 1 0 for a slow zoom In the current embodiment) then there 
is enough evidence of a zoom and the zoom detection shows a zoom state. 

Once the zoom detection shows a fast or slow zoom state, it will continue to 
10 show a zoom state until a zoom has not been detected for a numf>er of 

consecutive images (10 In the current embodiment). To smooth the transition, 
the zoom detection will show the possible zoom state for a few images (again, 
10 in the current embodiment) beibre returning to the nonsxtom state. 

15 If both a pan and a zoom are detected fbr a given image n, then a State 
Arbitration procedure 1 37 is used to decide which of these is the more 
important The prooedure works by assigning a priority to each of the ^tTes^ 
with the high^ priority one being acted upon» and the ottiers Ignored. The 
order used in the cnirrBnt embodiment is shown in Table 1 . In order of 

20 deoieasing priority: 



1. 


Fast Zoom 


2. 


Pan 


3. 


Slow Zoom 


4, 


Possible Pan 


5. 


Possible Zoom 


6. 


No Pan or Zoom 
detected. 



Table 1 



25 



Note that state 6 of Table 1 is selected by default if no other states are 
observed. 




The cam^ state 138 as predicted by module 106 Is into module 107 (86e 
Figure 2) where the stabilisation ofFiset (SO) tc be applied to image n is 
calcuiated. The other main input to module 107 is the error corrected 
GI\/IOec(r) 131 . The detected camera state is used generate a final ofl^et 
5 SO(n) 139 to be applied to the image n. This 1$ done as indicated in Table 2. 



Detect state 


Final Image stabilisation ofEset, SO(r} 


1 


SO(b) « rapid decay constant x SO(n -1} 


2 


SO{tt) - rapid decay constant x SO(w -1) ■♦- HPF(GMOec(w)) 


3 


SO(«) = decay constant x S0(n-1 ) + HPF(GMOec (n)) 


4.5or6 


SO(b) = decay constant x SO(«-1 ) + (GMOec («)) 



Table 2 



The decay constants In Table 2 are used to decay the accumulated image 
1 0 ofl^t over time. This slightly reduces the effect of stabiiisafton whilst 

improving the amount of image visible. If no camera motion is detected the 
decaying offset will ev^ituaiiy return the image to its Initial starting position, it 
is particular^ useful in the situation where carrora shake ceases t»jt the 
caiculafBd affset does not natum to zero. 

15 

Also, if an embodiment of the cunent invention is produced that does not iiave 
the capability to detect and con^ for pan or zoom movements (wAiidi may 
be done to Increase processing speed and Image throughput tor example), 
and the embodiment is Inadvertently used with a panning or zooming camera. 
20 then it allows the system to vKok, albeit virith slightly reduced fidelity durf ng 
panning or zooming operaltons. The rapid decay constant currently used Is 
0.735. and the standard decay constant used Is 0.98. 



25 



The high-para filter (HPF) operaUon used in Table 2 is the same as that done 
In the pan detection, deserved above. 
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The offset SO{n) 139 as derived above fs next appHed. along with the camera 
state 138 to the Image b to effect the stablHaatfon of the vWeo signal. This is 
done In t«o stages, as Indicated In Figure 10. The first stage 140 shifts the 
image n acconJing to the vector S0(») 139. This shifting of the Image may 
5 result In some bonlers of the image area not having any data, and hence 
being blank. The blank areas in successive Imagee may be of difPefent sizes, 
which would result in a flickering effect as the rapidly moving edges of the 
image sequence is presented to the output display device. The second stage 
of the display process therefore is to generate "dynamic" borders 141 that 
10 cover these blank areas 

The dynamic borders hide the rapidly moving edges of the stabilised image 
sequence. This is done by overlaying artificial black bonJere over the edges of 
the shifted image. These reduce the size of the visible image such thatthe 

15 rapidly moving edges are hidden. The borders continually adjust to show as 
much of the Image as possible withput showing the moving edges. The 
camera state, the stabiflsation offset SO(«) and a history of the SO values are 
used to determine the amount of border shown. When a pan or zoom Is 
occurring or there is little Image motion, the dynamic borders decay to show 

20 the edges of the Image. Typically the borcler will cover an area up to the 

maximum excursion of the Images detected within an offset history period of 
50 Images. 

The off&el history used for the border generation gains an entry for each 
25 image aooording to the camera slate: 

If a Fast Zoom Stete. Pan State. Slow Zoom state. Possible Pan State 
or Possible Zoom State te detected, then the value in the offset history for 
image n is set too. 

If a No Pan or Zoom Stete is detected, then the value In the offset history for 
30 image » is set to SO(n). 



To prevent the borders changing too rapidly, the borders are limited In their 
rate of change for each Image. The maximum change in the verticai dlrecUon 



is limited to 5 pixels per image and the maximum change in the horizontal 
direction l8 limited to 10 pixels per image. These values have been found to 
work well, but oilier valuee could be used. 

5 The shffted Image 142^ wl^ dynamic borders applied, is then converted to an 
analogue signal for replay on a monitor or recording to disic or tape. Of 
course, the digital signa! could also be saved to a computer disi< in any 
convenient format. 

10 Figure 1 1 shows the border generation in action. In a) a scene Is shown at 
which a video camera, mounted on an unstable platform, is pointed. Assume 
that the camera Is wobbling up and down. The field of view of the camera is 
the non-hashed portion 143. The large rectangle represents the larger scene 
144 as seen by the camera at any time during its movement over the scene 

15 144, over a period of a few seconds. It will be seen that the camera is, for this 
image, poirvHng towards the top of the scene 144, hence the lower portion of 
the ecene 145. represented by the hashed region, is not present. 

The image stabilisation system as disclosed herein, when laresented with a 
20 sequence of images of which Figure 11a was one. would tend to move the 
Image 143 in towards the centre of the display area of the replay devicei the 
upper and lower limit of which are here Indicated by dotted lines 1 51 . This 
would cause a gap at the top of the shifts fl'ame. When the dynamic border 
js generated, the border at the lop of the stabiileed Image is made at least as 
25 large as this movement, to hid^ thds gap. 

Figure lib) represents the image recorded by the camera a short Ijme period 
later, when the camera has moved due to Its wobble and Es now pointing 
towanjs the t^ottom 146 of the larger scene 144. Because of this the lop 147 
30 of the larger scene 144 has not been recorded In this frame. Again, the 
stabiiieetion routine would lend to move this image Into the centre of the 
display area as represented by doited lines is1, and the gap produced by this 
would also be covered when generafing the dynamic tKirder. 




As well as the above medianism causing blank areas at the top and ixittom of 
the stabilised image there are ajme areas of the larger ecene 144 that are 
visible in one Imase but not in anottier. For example, in Figure 1 la the top of 
a tree 148 can be eeen, whereas the ground cannot Likewise, In Figure lib, 
5 the ground 149 can be seen, but the top of the tree cannot tf the borders just 
covered up the blank areas, then there would stiil be visible a flickering region 
adjacent fliese borders caused by the scene being only visible at certain 
times. These are the rapidly changing edges referred to above. To hide 
these, the border is extended to cover ttiis region, the size of which is 
10 . detennined by examining the maximum excursion (given by the stabilisatton 
offset) of the image seen over the previous ^fty images. 

Figure 1 1 c shows the resultant stabilised Image, with borders 1 50, within the 
display area indicated by numeral 151 , generated as described above to 
1 5 cover the rapidly moving edges. 

. An alternative embodiment tacWesthe moving edge problem in a different 
manner. Here, where camera movement creates a blank area in an image, 
image information from a previous image is used to effiecflvely overlay the 

20 blank area. This Is done by creating a buffer image that comprises the current 
image as shifted by the offset as described above, that is written onto the 
previous lmage(s) in the buffer, such that it oven(wites only those parts where 
image data is present in the current image, and teaves untouched those parts 
• in the buffer that correspond to blank areas of the current Image, in this way. 

25 tile buffer image grows Into an image that Is larger than the display aree given 
by the bordara 1 51 , but is a composite of the cunent image and prevtous 
images. The buffer image wlll be the size of the larger scene 144. The part of 
this buffter Image fitting within the limits of the display as given by dotted lines 
1 51 is th^ output to the display or recording device, and thus ensures tiiat no 

30 dead areas or border space need be displayed . 




A further embodiment expands the size of image to be displayed by linearly 
scaiing it such that the image covers those parts that\WDutd othenwise be 
blanic due to the image ahifBi^ process. 



The sldlled person will be swvere that other embodlnnents within the scope of 
the Invention may be envisaged, and thus the invention should not be llmltBd 
to ttie embodiments as herein described. 
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1. A video image stabilisattan system for oonredion of camera fnoUon, that 
is arranged to receive one or more signals representative of a piurality of 

5 images from an Image source wherein, for an image « following at least an 
image (n -1) and an Image (» -2) the system is airanged to estimate a Global 
Motion Offset (GMO) value between Image n and a previous Image 
representative of the spatial sepai^on between the scene imaged In image n 
and the previous Image, and apply a conedlvc movement to the Image n 

10 based upon this GMO, characterised in that: 

the system Is arranged to estimate the GMO for the Image n with 
reference to a mask that represents a region or ragtone of the image n not to 
be considered In the GMO estimation, the region(s) being region{s) estimated 
as likely to mislead the estimation of the GMO. 



15 



2. A stabilisation system as claimed in claim 1 wherein the system is 
ananged to examine one or more locei regions of the Image a and 
corresponding local regions of a previous Image, and estimate a local motton 
oflset (LMO) representative of spatial separation between like features in 
20 con^pondlng locai regtons of the current and previous images, and if the. or 
each, mo Is greater than a given threshold, to set areaCs) of the mask that 
oonespond to this local region or regions to Indicate omission from the GMO 
estimation. 

25 3 A stablBsatJon system as claimed In claim 2 wherein the local regions 
comprise an anay of rectangular regions. 

4. A stabflteation system as claimed In any of claims 1 to 3 wherein the 
system 18 arranged to estimate the GMO of an Image representative of Image 

30 n but having a spatial resolution lower than image w. 

5. A stabilisation system as claimed In daim 4 wherein the system is 

. ananged to Iterate the estimation of the GMO on a plurality of images each 
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representative of image n, where each of the plurality of images ha$ a 
different spatial resolution. 

6. A sfabilisatfon system as dafmed in any of the above claims wherein 
5 the system is arranged to adjust the GMO if a stationary cemeina state 

detected, this state being indicated by means of a plunality of contiguous 
GMOs Induding the current GMO all being below a given threshold. 

7. A stabflisation system as dalmed in any of claims 2 to 6 wherein the 
1 0 system is arranged Id adjust flie GMO if intenflonal adjustment of the image 

source viewing direction (pan) or field of view (zoom) is detected. 

8. A stabilisation system as claimed in claim 7 wherein the system Is 
arranged to detect a pan of the image source by means of low-pass filtering 

15 GMO values from at least a sequence of previous images at a cut-off 

frequency lower than that expected from unintentional camera movements. 

9. A stabilisation system as claimed in claim 7 or claim 8 wherein a zoom 
is detected if a number x of LMOs examined for image » all show a direction of 

20 movement in towards a central region of the image n , or all show a direction 
of movement away from a central region of the nmage «, the number ;;c being 
greater than some given threshold. 

1 0. A stabilisation system as claimed in claim 9 wherein the threslwld 1$ 60% 
25 of those LMOs examined, and the nurhber jc is proportional to the absolute 

difference between the number of those LMOs examined shoving a direction 
of movement in towards a central region of the Image n , and those LMOs 
examined showing a dirsction of movement away from a central region of the 
image n. 

30 

11. A stebillsation system as daimed In claim 9 or daim 1 0 wherein the 
UMOs e)^mined ars taken from those local regions liiat are substantially 
adjacent tfie edge of Image n. 
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12. A stabilisation system as dalmed In any of the above claims wtierefn 
the system is ananged to generate a l»rder on at least one edge of the Image 
», the border being adjustable in size such that It covers any blank space 

5 between the edge of image n and the corresponding edge of a display area on 
\Mhlch the Image » Is displayed. 

13. A stabilisatton system as claimed m claim 12 wherein the system is 
arranged lo adjust the border size on at least one edge of the Image » such 

10 that It also covers an area on image n conesponding to blank space present 
on one or more prevk)U6 images. 

14. A stabilisation system as dalmed in dalm 12 when^n Ihe border 
generated by the system comprises of image data from on© or more previous 

15 images. 

r 

15.. A stabilisation system as claimed In any of claims 1 to 11 wherein the 
system is ananged to scale the image n. such that It covers any blank space 
between the edge of image n and the conresponding edge of a display area on 
20 whidi ihe image n is displayed. 

IB. A stabilisation system as dalmed In any of the above claims wherein 
anomalous pixels of the Image n are used to set corresponding pixels of the 
mask such that they are exduded from the estimation of the GMO. 

25 

17. A stabilisation system as claimed In daim 16 wherein the pixels above 
a thi^shoW in an Image comprising the absolute difference between the image 
n and a previous image m, both images n and m having had corrective 
nrtovenrients applied, are r^aided as anomalous. 

30 

18. A stabliisatbn system as datmed In ar^ of the above dalms wherein 
the system is ananged to multiply the calculated GMO. as adjusted in any 



other operation, by a decay constant factor lying between 0 and 1 before 
Shiftir^ the image n. 

19. A method of stabilising a present image relative to at least one pre^ous 
5 image where both cunent and previous Images are part of a sequence of 

video images represented by an elet^nic signal, oomprfsing the steps of: 
i. estimating a global motion offset (GMO) between the current 

and previous image representative of the spatial separation between the 

scene imaged in the current Image and that imaged in the previous Image; 
1Q and 

H. applying a conective movement to the current Image based 
upon the GMO; 

Gharacterissd in that 

a masit image is used in estimating the GMO, the masic image 
5 representing a region or i^gions of the current Image not to be considered in 
the GMO estimation, the regfon(s) being region(s) being estimated as iilcoly to 
mislead the estimation of the GMO. 

20. A method as daimed in daim 1 9 wherein the method furth^ includes 
0 the step of examining one or more local regions of the current image and 

corresponding [oca! regions of a previous Image, and estimating a local 
.. motion ofiiset (LMO) representing the spatial separation between iilce leaturss 
In corresponding local regions of the current and previous images, and if the, 
oreadi. LMO is greater than a given threshold, setting aFea(8) Gf the mask 
5 that correspond to this local region or regions to indicate omteslon from the 
GMO estimaUon. 

21 . A computer program designed to nin ori a computer and arranged to 
implement a video image stabilisation s^^tem, the system being arranged to 

3 receive as an input a digital signal representativa of a plurafity of images from 
an image source wherein, for an image n fcdtov^ng at least an image (n -1 } 
, and an image (n -2) the system Is arranged to ^mate a Global Motion Offset 
(GMO) value betvween image n and a previous image representeflve of the . 



^atiai separation between the soene imaged In image » and the previous 
image, and apply a conBdive movement to the Image « baaed upon this 
GMO, charac^sed inlfiat* 

the system Is arranged to estimate the QMO for the image n with 
reference to a masic that represents a region or regions of the image u not to 
be considered in the GMO estimation, the region(a) being regionCs) estimated 
as liKely to mislead the estimation of the GMO. 



;c .■„,...... 

Image Stabilisatton Systdm and Method 

A system fbr stabilising video signals generated fay a video camera which may 
be mounted In an unstable manner includes a digital processing means for 
manipuiation of each incoming image that attempts to overlay features of the 
current Image onto similar features of a previous Image. A maek Is used that 
prevents parts of the image that are likely to cause errors in the overlaying 
process from being used in the calculation of the required movement to be 
applied to the image* The mask may include areas where smalt movements 
of the Image have been detected, and may also include bt&bs where image 
anomaOes Including excess rtoise have been detected. The invention also 
discloses means for dealing with wanted movements of the camera, such as 
pan or zoom, and also disclosed means for dealing with the edges of vtdeo 
signals as processed by the system. A method is also disclosed. 

(Figure 2) 
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